## [1] "Excluded 31 participants based on catch-trial performance."
We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.
## [1] "Excluded 1 participants based on random responses."
We use the AUC function with the splines method to directly compute the AUC.
t-test and regression model with control variables:
##
## Two Sample t-test
##
## data: aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = 2.9171, df = 142, p-value = 0.004108
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 4.110558 21.393403
## sample estimates:
## mean of x mean of y
## 18.011474 5.259493
##
## Call:
## lm(formula = auc_diff ~ cond + test_order + first_speaker_type +
## confident_speaker, data = rbind(aucs.cautious, aucs.confident))
##
## Residuals:
## Min 1Q Median 3Q Max
## -75.675 -13.841 0.159 16.143 72.807
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 15.530 4.879 3.183 0.00180 **
## condconfident (probably-biased) -12.752 4.387 -2.907 0.00425 **
## test_orderreverse -2.275 4.392 -0.518 0.60527
## first_speaker_typeconfidentfirst 5.716 4.395 1.300 0.19562
## confident_speakerconfidentm 1.209 4.395 0.275 0.78372
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 26.32 on 139 degrees of freedom
## Multiple R-squared: 0.06997, Adjusted R-squared: 0.04321
## F-statistic: 2.615 on 4 and 139 DF, p-value: 0.03789